Apparatus method and system for imaging

ABSTRACT

There is provided in accordance with some embodiments of the present invention an optical assembly including a set of optical paths, wherein two or more optical paths receive an image from a common surface. The optical paths may direct received images onto a common image sensor generating a complex multidimensional data set, an image processing block may extrapolate each of the subset of optical paths printed on the image sensor and may generate a multidimensional data set based on the collected images.

FIELD OF THE INVENTION

The present invention relates generally to the field of imaging. Morespecifically, the present invention relates to an apparatus, method andsystem having multiple optical paths for acquiring one or more imagesfrom a scene and producing one or more data sets representative ofvarious aspects of the scene, including depth (i.e. ThreeDimensional—3D) information.

BACKGROUND

Conventional 3D-stereoscopic photography typically employs twin camerashaving parallel optical axes and a fixed distance between their alignedlenses. These twin cameras generally produce a pair of images whichimages can be displayed by any of the known in the art techniques forstereoscopic displaying and viewing. These techniques are based, ingeneral, on the principle that the image taken by a right lens isdisplayed to the right eye of a viewer and the image taken by the leftlens is displayed to the left eye of the viewer.

For example, U.S. Pat. No. 6,906,687, assigned to Texas InstrumentsIncorporated, entitled “Digital formatter for 3-dimensional displayapplications” discloses a 3D digital projection display that uses aquadruple memory buffer to store and read processed video data for bothright-eye and left-eye display. With this formatter video data isprocessed at a 48-frame/sec rate and readout twice (repeated) to providea flash rate of 96 (up to 120) frames/sec, which is above the displayflicker threshold. The data is then synchronized with a headset orgoggles with the right-eye and left-eye frames being preciselyout-of-phase to produce a perceived 3-D image.

Spherical or panoramic photographing is traditionally done either by avery wide-angle lens, such as a “fish-eye” lens, or by “stitching”together overlapping adjacent images to cover a wide field of vision, upto fully spherical fields of vision. The panoramic or spherical imagesobtained by using such techniques can be two dimensional images orstereoscopic images, giving to the viewer a perception of depth. Theseimages can also be computed as three dimensional (3D-depth) images interms of computing the distance of every pixel in the image from thecamera using known in art passive methods such as triangulation methods,semi active methods or active methods.

For example, U.S. Pat. No. 6,833,843, assigned to Tempest MicrosystemsIncorporated, teaches an image acquisition and viewing system thatemploys a fish-eye lens and an imager such as, a charge coupled device(CCD), to obtain a wide angle image, e.g., an image of a hemisphericalfield of view.

Reference is also made to applicant's co-pending U.S. patent applicationSer. No. 10/416,533, filed Nov. 28, 2001, the contents of which arehereby incorporated by reference. The application teaches an imagingsystem for obtaining full stereoscopic spherical images of the visualenvironment surrounding a viewer, 360 degrees both horizontally andvertically. Displaying the images by means suitable for stereoscopicdisplaying, gives the viewers the ability to look everywhere aroundthem, as well as up and down, while having stereoscopic depth perceptionof the displayed images. The disclosure teaches an array of cameras,wherein the lenses of the cameras are situated on a curved surface,pointing out from C common centers of said curved surface. The capturedimages are arranged and processed to create a pair of stereoscopic imagepairs, wherein one image of said pair is designated for the observer'sright eye and the second image for his left eye, thus creating a threedimensional perception.

3D Depth Images Using Active Methods

Active methods may intentionally project high-frequency illuminationinto the scene in order to construct 3D measurement of the image. Forexample, 3DV systems Incorporated (http://www.3dvsystems.com/) providesthe ZCam™ camera which captures, in real time, the depth value of eachpixel in the scene in addition to the color value, thus creating a depthmap for every frame of the scene by grey level scaling of the distances.The Zcam™ camera is a uniquely designed camera which employs a lightwall having a proper width. The light wall may be generated, forexample, as a square laser pulse. As the light wall hits objects in aphotographed scene it is reflected towards the ZCam™ camera carrying animprint of the objects. The imprint carries all the information requiredfor the reconstruction of the depth maps.

3D Depth Images Using Passive Methods

Using passive methods, for example stereo algorithms may attempt to findmatching image features between a pair of images about which nothing isknown a priori. Passive methods for depth construction may usetriangulation techniques that make use of at least two known sceneviewpoints. Corresponding features are identified, and rays areintersected to find the 3D position of each feature.

Corresponding features may be identified over space and time, asexemplified in FIG. 1, the basic principle of space-time stereo is thattraditional stereo matches vectors in the spatial or image domain todetermine correspondence between pixels in a single pair of images for astatic moment in time.

In temporal stereo, using multiple frames across time, we match a singlepixel from the first image against the second image. Rather thanincreasing a vector by considering a neighborhood in the spatialdirection, it is possible to increase a vector in the temporaldirection.

Space-time stereo adds a temporal dimension to the neighborhoods used inthe spatial matching function. Adding temporal stereo, using multipleframes across time, we match a single pixel from the first image againstthe second image. This can also be done by matching space-timetrajectories of moving objects, in contrast to matching interest points(corners), as done in regular feature-based image-to-image matchingtechniques. The sequences are matched in space and time by enforcingconsistent matching of all points along corresponding space-timetrajectories, also obtaining sub-frame temporal correspondence(synchronization) between two video sequences. For example “Tidex”(www.tidexsystems.com/products.htm) extracts 3D depth information from a2D video sequence of a rigid structure and modeling it as a rigid 3Dmodel. Another example is the “SOS technology”(www.hoip.jp/ENG/sosENG.htm,www.viewplus.co.jp/products/sos/astro-e.html) Stereo Omni-directionalSystem (SOS), that simultaneously acquires images and 3D information onevery direction using 20 sets of 3 cameras each, 60 cameras in total asshown in FIG. 2, applying for various fields, such as scene sensing androbot vision.

Semi Active Methods

Semi active methods intentionally project high-frequency illuminationinto the scene to aid in determining good correspondences, significantlyimproving performance in areas of little texture in order to construct3D measurement of the image. Constructing easily identifiable featuresin order to minimize the difficulty involved in determiningcorrespondence, illumination such as laser scanning and structuredlight, carrying various colors, patterns or a certain threshold overtime. The illumination means can comprise one or more laser sources,such as a small diode laser, or other small radiation sources forgenerating beams of visible or invisible light in a set of points in thearea of the lenses to form a set of imprinted markers in the capturedimages. Said set of imprinted markers are identified and enable passivemethods to facilitate image processing in accordance with known in artpassive processing methods. For example structured light scanning wherea known set of temporal patterns (the structured light patterns) areused for matching. These patterns induce a temporal matching vector.Structured light is a special case of space-time stereo, with matchingin the temporal domain. Another example would be laser scanning, where asingle camera and a laser scanner sweeps across a scene. A plane oflaser light is generated from a single point of projection and is movedacross the scene. At any given time, the camera can see the intersectionof this plane with the object. Both spatial and temporal domain laserscanners have been built for that purpose.

Holographic-Stereogram

Production of holographic stereogram from two-dimensional photographs isan established technique, first described by De Bietetto (1969). Unliketraditional holograms, holographic stereograms consist of informationrecorded from a number of discrete viewpoints. Laser-illuminated displayholography, developed in 1964 by Leith and Upatnieks, was the firsttruly high quality three-dimensional display medium. Hologram isburdened by the fact that it is not only a display but also a recordingmedium. Holographic recording must be done in monochromatic, coherentlight, and requires that the objects being imaged remain stable towithin a fraction of a wavelength of light. These requirements havehindered holography from gaining widespread use. In addition, the amountof optical information stored in a hologram makes the computation ofholographic patterns very difficult.

A holographic stereogram records a relatively large number of viewpointsof an object and may use a hologram to record those viewpoints andpresent them to a viewer. The information content of the stereogram isgreatly reduced from that of a true hologram because only a finitenumber of different views of the scene are stored. The number of viewscaptured can be chosen based on human perception rather than on thestorage capacity of the medium. The capturing of the viewpoints for thestereogram is detached from the recording process; image capture isphotographic and optically incoherent, so that images of natural sceneswith natural lighting can be displayed in a stereogram. The input viewsfor traditional stereograms are taken with ordinary photographic camerasand can be synthesized using computer graphic techniques. Using recentlydeveloped true color holographic techniques, extremely high quality,accurate, and natural-looking display holograms can be produced.Horizontal Parallax Only (HPO) stereograms provide the viewer with mostof the three-dimensional information about the scene, with a greatlyreduced number of camera viewpoints and holographic exposures then thefull-parallax stereograms. The principles however, apply equally to bothHPO and full-parallax stereograms.

There are three stages to the stereogram creation: photographic capture,holographic recording, and final viewing. The holographic stereogram isa means of approximating a continuous optical phenomenon in a discreteform. In display holo-stereography, the continuous three-dimensionalinformation of an object's appearance can be approximated by arelatively small number of two-dimensional images of that object. Whilethese images can be taken with a photographic camera or synthesizedusing a computer, both capture processes can be modeled as if a physicalcamera was used to acquire them. The photographic capture, theholographic recording, and the final viewing geometries all determinehow accurately a particular holographic stereogram approximates acontinuous scene.

There are a number of stereogram capturing methods. For example, somemay require a Holographic exposure setup using a single holographicplate comprised of a series of thin vertical slit holograms exposed onenext to the other across the plate's horizontal extent. Each slit isindividually exposed to an image projected onto a rear-projection screensome distance away from the plate. Once the hologram is developed, eachslit forms an aperture through which the image of the projection screenat the time of that slit's exposure can be seen. The images projectedonto the screen are usually views of an object captured from manydifferent viewpoints. A viewer looking at the stereogram will see twodifferent projection views through two slit apertures, one through eacheye. The brain interprets the differences between the two views asthree-dimensional information. If the viewer moves side to side,different pairs of images are presented, and so the scene appears togradually and accurately change from one viewpoint to the next tofaithfully mimic the appearance of an actual three-dimensional scene.Some methods may use multiple recentered camera arrays.

The actual projected image, whose extent is defined by the projectionframe, is the visible sub region of the projection screen in anyparticular view. The projection screen itself is a sub region of a planeof infinite extent called the projection plane.

The projection screen directly faces the holographic plate and slitmechanism or the camera. The viewer interprets the two imagesstereoscopically. This binocular depth cue is very strong; horizontalimage parallax provides most of the viewer's depth sense. Using theobservation that a stationary point appears to be at infinity as alandmark, the correct camera geometry needed to accurately capture athree-dimensional scene can be inferred. To appear at infinity, then, anobject point must remain at the same position in every camera view. Thisconstraint implies that the camera should face the same direction,straight ahead, as each frame is captured. The camera moves along atrack whose position and length correspond to the final stereogramplate. The camera takes pictures of a scene from viewpoints thatcorrespond to the locations of the stereogram's slits. The plate isplanar, so the camera track must be straight, not curved. The cameramust be able to image the area corresponding to the projection frameonto its film; thus, the frame defines the cross section of the viewingpyramid with its apex located at the camera's position. Because theprojection frame bounds the camera's image, the size of the projectionframe and its distance from the slit determine the angle of view of theimage and thus the maximum (and optimal) focal length of the camera'slens. The film plane of the stereogram capture camera is always parallelto the plane of the scene that corresponds to the projection plane (thecapture projection plane) in order to image it without geometricdistortions onto the focal plane of the lens.

A stereogram exposure geometry is well suited for objects far from thecamera when the image of the object wanders little from frame to frame,always remaining in the camera's field of view and thus always visibleto the stereogram viewer. However, distant objects are seldom the centerof interest in three-dimensional images because the differentperspectives captured over the view zone have little disparity and, as aresult, convey little sense of depth. Objects at more interestinglocations, closer to the camera, wander across the frame from one cameraview to the next and tend to be vignetted in the camera's image ateither or both extremes of the camera's travel. The solution to theproblem is to alter the capture camera to always frame the object ofinterest as it records the photographic sequence. Effectively, thischange centers the object plane in every camera frame so that it remainsstationary on the film from view to view. Object points in front of orbehind the stationary plane will translate horizontally from view toview, but at a slower rate than they would in a simple camerastereogram. Altering the camera geometry requires changes in theholographic exposure geometry needed to produce undistorted images. Theprojection screen is no longer centered in front of the slit apertureduring all exposures. Instead, the holographic plate holder isstationary and the slit in front of it moves from exposure to exposure.Thus, the projection frame is fixed in space relative to the plate forall exposures, rather than being centered in front of each slit duringeach exposure. In this geometry, called the “recentered camera”geometry, only one projection frame position exists for all slits. Ineffect, as the viewer looks at the final stereogram, the projectionframe no longer seems to follow the viewer but instead appearsstationary in space. If an image of the object plane of the originalscene remains stationary on the projection screen, then, the objectplane of the original scene and the projection plane of the finalhologram will lie at the same depth.

One type of camera which may take pictures for this type of stereogramis called a recentering camera. Recall that in the simple camera imagecapture, the image of a nearby object point translated across thecamera's film plane as the camera moved down its track taking pictures.In a recentering camera, the lens and the film back of the camera canmove independently from each other, so the film plane can be translatedat the same rate as the image of the object of interest. The film andimage move together through all frames, so just as desired the imageappears stationary in all the resulting images. A view camera with a“shifting” or “shearing” lens provides this type of recentering. Thelens of the camera must be wide enough to always capture the fullhorizontal extent of the object plane without vignetting the image atextreme camera positions. A correspondence must exist between the cameracapture and the holographic exposure geometries. In the recenteringcamera system, the necessary translation of the camera's lens addsanother constraint that must be maintained. A point in the middle of theobject plane must always be imaged into the middle of the film plane,and must always be projected onto the middle of the projection frame.The angle subtended by the object frame as seen from the camera mustequal to the angle subtended by the projection frame as seen from theslit. If, for example, the focal length of the lens of the taking camerais changed, the amount of lens translation required and the size of theholographic projection frame would also have to be adjusted.

To summarize, there are two common methods of producing adistortion-free holographic stereogram from a sequence of images: thefirst in which the projection frame is located directly in front of theslit during each exposure and the plate translates with respect to it(the “simple camera” geometry), and the second in which the screen iscentered in front of the plate throughout all the exposures and the slitmoves from one exposure to the next (the “recentering camera” geometry).The first method has the advantage that the camera needed to acquire theprojectional images is the easier to build, but the input frames tend tovignette objects that are close to the camera. The second methodrequires a more complicated camera, but moves the plane of the imagewhere no vignetting occurs from infinity to the object plane. The cameracomplexity of this method is less of an issue if a computer graphicscamera rather than a physical camera is used.

The projection frame in a recentered camera stereogram forms a “window”of information in space, fixed with respect to the stereogram andlocated at the depth of the projection plane. The usefulness of thisfixed window becomes important when the slit hologram is opticallytransferred in a second holographic step in order to simplify viewing.To maintain the capture-recording-viewing correspondence in anystereogram, the viewer's eyes must be located at the plane of the slithologram. When the stereogram is a physical object, the viewer's facemust be immediately next to a piece of glass or film. However, aholographic transfer image can be made so as to project a real image ofthe slit master hologram out into space, allowing the viewer to beconveniently positioned in the image of the slits, hologram of a realobject.

In the simple camera stereogram, the images of the projection framesthat the slits of the master project to the transfer during masteringare shifted with respect to each other because each frame image iscentered directly in front of its slit. Thus, the frames cannotcompletely overlap each other. In the case of the recentered camerastereogram, however, all images of the projection frames preciselyoverlap on the projection plane. When the transfer hologram is made, theposition of the transfer plate on the projection plane determines whatwindow of that plane will be visible to the viewer. In the recenteringcamera stereogram, this window is clearly defined by the projectionframe: all information from all slits overlaps there, with no datawasted off the frame's edge. In the simple-camera case, some informationfrom every slit (except the center one) will miss the transfer's frameand as a result will never be visible to the viewer.

A holographic stereogram can of course be cylindrical, for all-roundview. In this case, the transparencies can be made by photographing arotating subject from a fixed position. If the subject articulates aswell, each frame is a record of a particular aspect at a particulartime. A rotating cylindrical holographic stereogram made from successiveframes of movie film can then show an apparently three-dimensionaldisplay of a moving subject.

Understanding the optical effects of moving the viewer to a differentview distance requires another means of optical analysis called raytracing. While wavefront analysis is useful when determining the smallchanges in the direction of light that proved significant in thestereogram's wavefront approximation, ray tracing's strength is inillustrating the general paths of light from large areas, overlookingsmall differences in direction and completely omitting phase variations.Ray tracing can be used to determine the image that each camera along atrack sees, and thus what each projection screen should look like wheneach slit is exposed. It also shows what part of the projection screenof each slit is visible to a viewer at any one position. Distortion-freeviewing requires that the rays from the photographic capture step andthe viewing step correspond to each other.

Digital Camera Sensors

A digital camera uses a sensor array (e.g. Charge Coupled Devices—CCD)comprised of millions of tiny electroptical receptors that enablesdigitization or digital printing of an optical image. The basicoperation of the sensor is to convert light into electrons. When youpress your camera's shutter button and the exposure begins, each ofthese receptors is a “photosite” which collects and uses photons toproduce electrons. Once the exposure finishes, the camera closes each ofthese photosites, and then tries to assess how many photons fell intoeach photosite by measuring the number of electrons. The relativequantity of photons which fell onto each photosite is then sorted intovarious intensity levels, whose precision may be determined by a bitdepth (e.g. twelve bits for each photosite in the image results in aresolution level of (2¹²)=4095 possible values). The ability to generatea serial data stream from a large number of photosites enables the lightincident on the sensor to be sampled with high spatial resolution in acontrolled and convenient manner. The simplest architecture is a linearsensor, consisting of a line of photodiodes adjacent to a single CCDreadout register.

A common clocking technique is the 4-phase clocking system which uses 4gates per pixel. At any given time, two gates act as barriers (no chargestorage) and two provide charge storage.

In order to cover the entire surface of the sensor. Digital cameras maycontain “microlenses” above each photosite to enhance theirlight-gathering ability. These lenses are analogous to funnels whichdirect photons into the photosite where the photons would have otherwisebeen unused. Well-designed microlenses can improve the photon signal ateach photosite, and subsequently create images which have less noise forthe same exposure time.

Each photosite is unable to distinguish how much of each color hasfallen in, so the above illustration would only be able to creategrayscale images. To capture color images, each photosite has to have afilter placed over it which only allows penetration of a particularcolor of light. Virtually all current digital cameras can only captureone of the three primary colors in each photosite, and so they discardroughly ⅔ of the incoming light. As a result, the camera has toapproximate the other two primary colors in order to have informationabout all three colors at every pixel in the image. The most common typeof color filter array is called a “Bayer array”, or Bayer Filter.

Bayer Filter

A Bayer filter mosaic is a color filter array (CFA) for arranging RGBcolor filters, as shown in FIG. 3, on a square grid of photo sensors.The term derives from the name of its inventor, Bryce Bayer of EastmanKodak, and refers to a particular arrangement of color filters used inmost single-chip digital cameras (mostly CCD, as apposed to CMOS).

A Bayer array consists of alternating rows of red-green and green-bluefilters. The Bayer array contains twice as many green as red or bluesensors. These elements are referred to as samples and afterinterpolation become pixels. Each primary color does not receive anequal fraction of the total area because the human eye is more sensitiveto green light than both red and blue light. Redundancy with greenpixels produces an image which appears less noisy and has finer detailthan could be accomplished if each color were treated equally. This alsoexplains why noise in the green channel is much less than for the othertwo primary colors. Digital cameras that use different digital sensorsuch as CMOS for example, capture all three colors at each pixellocation. The RAW output of Bayer-filter cameras is referred to as aBayerPattern image. Since each photosite is filtered to record only oneof the three colors, two-thirds of the color data is missing from each.A Demosaicing algorithm is used to interpolate a set of complete red,green, and blue values for each point, to make an RGB image. Manydifferent algorithms exist.

Demosaicinq

Demosaicing is the process of translating an array of primary colors(such as Bayer array) into a final image which contains full colorinformation (RGB) at each point in the image which may be referred to asa pixel. A Demosaicing algorithm may be used to interpolate a completeimage from the partial raw data that one typically receives from thecolor-filtered CCD image sensor internal to a digital camera. The mostbasic idea is to independently interpolate the R, G and B planes. Inother words, to find the missing green values, neighboring green valuesmay be used, to find the missing blue values neighboring blue pixelsvalues may be used, and so on for red pixel values. For example, forlinear interpolation, to obtain the missing green pixels, calculate theaverage of the four known neighboring green pixels. To calculate themissing blue pixels, proceed in two steps. First, calculate the missingblue pixels at the red location by averaging the four neighboring bluepixels. Second, calculate the missing blue pixels at the green locationsby averaging the four neighboring blue pixels. The second step isequivalent to taking ⅜ of each of the closest pixels and 1/16 of fournext closest pixels. This example of interpolation introduces aliasingartifacts. Improved methods exist to obtain better interpolation.

RAW Image Format

The RAW file format is digital photography's equivalent of a negative infilm photography: it contains untouched, “raw” photosite informationstraight from the digital camera's sensor. The RAW file format has yetto undergo Demosaicing, and so it contains just one red, green, or bluevalue at each photosite. The image must be processed and converted to anRGB format such as TIFF, JPEG or any other known in the art compatibleformat, before it can be manipulated. Digital cameras have to makeseveral interpretive decisions when they develop a RAW file, and so theRAW file format offers you more control over how the final image isgenerated. A RAW file is developed into a final image in several steps,each of which may contain several irreversible image adjustments. Onekey advantage of RAW is that it allows the photographer to postponeapplying these adjustments—giving more flexibility to the photographerto later control the conversion process, in a way which best suits eachimage.

Demosaicing and white balance involve interpreting and converting theBayer array into an image with all three colors at each pixel, and occurin the same step. RAW image is then converted into 8-bits per channel,and may be compressed into a JPEG based on the compression settingwithin the camera. RAW image data permits much greater control of theimage. White balance and color casts can be difficult to correct afterthe conversion to RGB is done. RAW files give you the ability to set thewhite balance of a photo *after* the picture has been taken—withoutunnecessarily destroying bits. Digital cameras actually record eachcolor channel with more precision than the 8-bits (256 levels) perchannel used for JPEG images. Most \cameras files have a bit depth of 12or 14 bits in precision per color channel, providing several times morelevels than could be achieved by using an in-camera JPEG. This may allowfor exposure errors to be corrected. RAW may use different RGBconversion algorithms then the one coded into the camera.

RAW file formats are proprietary, and differ greatly from onemanufacturer to another, and sometimes between cameras made by onemanufacturer. In 2004 Adobe Systems published the Digital NegativeSpecification (DNG), which is intended to be a unified raw format. As of2005, a few camera manufacturers have announced support for DNG.

SUMMARY OF THE INVENTION

There is provided in accordance with some embodiments of the presentinvention an optical assembly including a set of optical paths. Anoptical path, according to some embodiments of the present invention,may include a lens and a diaphragm structure, wherein a given opticalpath's lens and diaphragm structure may be adapted to receive and/orcollect optical image information corresponding to one or more featuresof a given projection plain (e.g. visible features within the givenprojection plane). Two or more optical paths from the set of opticalpaths may receive and/or collect optical image information from a commonprojection plane.

According to some embodiments of the present invention, the projectionplane may be flat. According to further embodiments of the presentinvention, the projection plane may be any other shape, includingspherical, cylindrical or any other projection surface which may bedefined using optical elements such as lenses.

According to a further embodiment of the present invention, each of thetwo or more optical paths may direct their respective received/collectedoptical image information onto an image sensor, which image sensor maybe adapted to convert the optical image information into an image dataset correlated to the optical image information (e.g. a digital imageframe representing the collected optical image). According to furtherembodiments of the present invention, an image sensor may be adapted toproduce a series of image data sets (i.e. series of digital imageframes), wherein each image data set is representative of opticalinformation received/collected over a given period of time (e.g. 30milliseconds).

According to some embodiments of the present invention, each of the twoor more optical paths may direct its respective received/collectedoptical image information onto a separate image sensor, while accordingto further embodiments of the present invention, the two or more opticalpaths may direct their received/collected optical image information ontoa common image sensor. According to embodiments of the present inventionwhere a common image sensor is used by multiple optical paths, eachoptical path may either direct its respective collected image onto aseparate portion of the image sensor, or two or more optical paths maydirect their respective collected images onto overlapping segments onthe image sensor. According to even further embodiments of the presentinvention, two or more optical paths may direct their respectivecollected images onto a common segment of the image sensor, therebyoptically encoding the images.

According to embodiments of the present invention whereinreceived/collected optical image information from each of multipleoptical paths is directed to a separate segment of a common opticalsensor, image data produced from each separate optical sensor's segmentmay be considered a separate image data set (e.g. frame). According toembodiments of the present invention where two or more collected opticalimages are directed to a common segment on an image sensor (e.g.substantially the entire active/sensing area of the sensor), severalmethods may be used to produce a separate image data set associated witheach of the directed optical images, said methods including: (1) timedomain multiplexing, and (2) an encoding/decoding function.

According to embodiments of the present invention employing time domainmultiplexing, an optical shutter (e.g. Liquid Crystal Display shutter)may be included as part of each relevant optical path. Time domainmultiplexing may be achieved by opening only one given optical path'sshutters during a given period, which given period is within anacquisition period during which the image sensor is to produce imagedata associated with the given optical path. By opening the shutter ofeach of the relevant optical paths in correlation with the imagesensor's acquisition periods, multiple image data sets (i.e. imageframes) may be produced, wherein each image data set may be associatedwith a separate optical path.

According to embodiments of the present invention where optical imageinformation received/collected by each of a set of optical paths iscombined (i.e. optically encoded) on a common surface segment of theimage sensor during one or more sensor acquisition periods, the opticalsensor may produce a composite image data set that may includeinformation relating to an encoded composite of the two or morecollected optical images. An encoding/decoding function method oralgorithm in accordance with some embodiments of the present inventionmay be used to encode the two or more collected optical images anddecode the composite image data set into two or more separate image datasets, wherein each of the separate image data sets may be associatedwith and may represent collected optical image information from a singleoptical path.

According to some embodiments of the present invention, there isprovided an image data processing block, implemented either on adedicated data processor or on a programmable generable purposeprocessor. One or multiple image processing algorithms may beimplemented or executed via the processing block. For example, inaccordance with some embodiments of the present invention, the dataprocessing block may be adapted to perform a decoding function of acomposite image data set, and in accordance with further embodiments ofthe present invention, the processing block may be adapted to combinetwo or more collected images into a multidimensional (e.g. fourdimensional) data set, wherein the multidimensional data set may includeimage data representing various features of the common projection plain.

According to further embodiments of the present invention, there may beprovided an image extrapolation block, implemented either on a dedicateddata processor or on a programmable generable purpose processor. Theimage extrapolation block may extrapolate either from themultidimensional data set, from the originally acquired image data sets(i.e. frames) or from the encoded data set of the encoding/decodingfunction one or more types of derived data sets, wherein eachextrapolated data set type may be associated with one or more featuresof the common projection plain from which the two or more optical pathscollected optical image information. Extrapolated data set types mayinclude: (1) a depth map (i.e. z-channel or depth information of everypixel's-point in the common projection plain), (2) a holographicstereogram image of the common projection plain, (3) a stereo image ofthe common projection plain, and (4) one or more two-dimensional images,where each image may also be an approximated virtual view point of thecommon projection plain.

According to some embodiments of the present invention, each of theoptical paths may include (1) a fixed mirror, (2) a fixed lens and (3) afixed diaphragm structure. According to further embodiments of thepresent invention, the lenses and mirrors of two or more optical pathsmay be functionally associated (e.g. synchronized). According to yet afurther embodiment of the present invention, the diaphragms on the twoor more given optical paths having functionally associated lenses andmirrors may be adapted to adjust their configuration (e.g. aperture sizeand shape) so as to maintain a common projection plain between the givenoptical paths when the focus on the synchronized lenses is changed.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter regarded as the invention is particularly pointed outand distinctly claimed in the concluding portion of the specification.The invention, however, both as to organization and method of operation,together with objects, features, and advantages thereof, may best beunderstood by reference to the following detailed description when readwith the accompanying drawings in which:

FIG. 1 shows a series of graphs depicting the basic principles of spacetime stereo;

FIG. 1B shows a geometric diagram illustrating the basic principles andformulas relating to generating a disparity map using two sensors;

FIG. 1C shows a geometric diagram illustrating the basic principles andformulas relating to generating a disparity map using multiple sensors;

FIG. 2 shows a video camera array that may be used to extrapolate 360degree depth information using a passive method;

FIG. 3 shows an example of a Bayer filter;

FIG. 4A shows an optical assembly according to some embodiments of thepresent invention;

FIG. 4B shows a diagrammatic representation of an optical assemblyaccording to some embodiments of the present invention where opticalimage information from two separate optical paths are projected ontoseparate areas of a common image sensor;

FIG. 4C shows a diagrammatic representation of an optical assemblyaccording to some embodiments of the present invention where opticalimage information from two separate optical paths is projected onto acommon area of a common image sensor;

FIG. 5A shows a symbolic diagram of an optical assembly and imageprocessing system, according to some embodiments of the presentinvention, producing a series of digital image data sets (e.g. frames);

FIG. 5B shows a symbolic diagram depicting multiple images beinggenerated from a multidimensional image data set according to someembodiments of the present invention;

FIG. 5C shows a symbolic diagram depicting various image related datasets being derived from the multidimensional image data set according tosome embodiments of the present invention;

FIG. 6A shows a block diagram representing image processingelement/modules according to some embodiments of the present invention;

FIG. 6B shows a flow diagram including the steps of producing one ormore multidimensional image data sets from two or more acquired imageframes according to some embodiments of the present invention;

FIG. 6C shows a flow diagram including the steps of producing one ormore multidimensional image data sets from an optically encoded imageframe according to some embodiments of the present invention.

FIGS. 7A, 7B and 7C show exemplary filters according to some embodimentsof the present invention.

FIG. 8 shows an exemplary filter according to some embodiments of thepresent invention.

FIG. 9A shows two images which are used as an input to an algorithmaccording to some embodiment of the present invention.

FIGS. 9B and 9C show the outcome of image processing, which is performedin accordance with some embodiment of the present invention.

FIG. 10 shows a flow diagram including the steps of an algorithm inaccordance with some embodiments of the present invention.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity. Further, where consideredappropriate, reference numerals may be repeated among the figures toindicate corresponding or analogous elements.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the invention.However, it will be understood by those skilled in the art that thepresent invention may be practiced without these specific details. Inother instances, well-known methods, procedures, components and circuitshave not been described in detail so as not to obscure the presentinvention.

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “computing”,“calculating”, “determining”, or the like, refer to the action and/orprocesses of a computer or computing system, or similar electroniccomputing device, that manipulate and/or transform data represented asphysical, such as electronic, quantities within the computing system'sregisters and/or memories into other data similarly represented asphysical quantities within the computing system's memories, registers orother such information storage, transmission or display devices.

Embodiments of the present invention may include apparatuses forperforming the operations herein. This apparatus may be speciallyconstructed for the desired purposes, or it may comprise a generalpurpose computer selectively activated or reconfigured by a computerprogram stored in the computer. Such a computer program may be stored ina computer readable storage medium, such as, but is not limited to, anytype of disk including floppy disks, optical disks, CD-ROMs,magnetic-optical disks, read-only memories (ROMs), random accessmemories (RAMs) electrically programmable read-only memories (EPROMs),electrically erasable and programmable read only memories (EEPROMs),magnetic or optical cards, or any other type of media suitable forstoring electronic instructions, and capable of being coupled to acomputer system bus.

The processes and displays presented herein are not inherently relatedto any particular computer or other apparatus. Various general purposesystems may be used with programs in accordance with the teachingsherein, or it may prove convenient to construct a more specializedapparatus to perform the desired method. The desired structure for avariety of these systems will appear from the description below. Inaddition, embodiments of the present invention are not described withreference to any particular programming language. It will be appreciatedthat a variety of programming languages may be used to implement theteachings of the inventions as described herein.

There is provided in accordance with some embodiments of the presentinvention an optical assembly including a set of optical paths. Anoptical path according to some embodiments of the present invention mayinclude a lens and a diaphragm structure, where a given optical path'slens and diaphragm structure may be adapted to receive and/or collectoptical image information corresponding to one or more features of agiven projection plain (e.g. visible features within the givenprojection plane). Two or more optical paths from the set of opticalpaths may receive and/or collect optical image information from a commonprojection plane.

According to some embodiments of the present invention, the projectionplane may be flat. According to further embodiments of the presentinvention, the projection plane may be any other shape, includingspherical, cylindrical or any other projection surface which may bedefined using optical elements such as lenses or mirrors.

According to a further embodiment of the present invention, each of thetwo or more optical paths may direct their respective received/collectedoptical image information onto an image sensor, which image sensor maybe adapted to convert the optical image information into an image dataset correlated to the optical image information (e.g. a digital imageframe representing the collected optical image). According to furtherembodiments of the present invention, an image sensor may be adapted toproduce a series of image data sets (i.e. series of digital imageframes), wherein each image data set is representative of opticalinformation received/collected over a given period of time (e.g. 30milliseconds).

According to some embodiments of the present invention, each of the twoor more optical paths may direct its respective received/collectedoptical image information onto a separate image sensor, while accordingto further embodiments of the present invention, the two or more opticalpaths may direct their received/collected optical image information ontoa common image sensor. According to embodiments of the present inventionwhere a common image sensor is used by multiple optical paths, eachoptical path may either direct its respective collected image onto aseparate portion of the image sensor, or two or more optical paths maydirect their respective collected images onto overlapping segments onthe image sensor. According to even further embodiments of the presentinvention, two or more optical paths may direct their respectivecollected images onto a common segment of the image sensor, therebyoptically encoding the images.

According to embodiments of the present invention whereinreceived/collected optical image information from each of multipleoptical paths is directed to a separate segment of a common opticalsensor, image data produced from each separate optical sensor's segmentmay be considered a separate image data set (e.g. frame). According toembodiments of the present invention where two or more collected opticalimages are directed to a common segment on an image sensor (e.g.substantially the entire active/sensing area of the sensor), severalmethods may be used to produce a separate image data set associated witheach of the directed optical images, said methods including: (1) timedomain multiplexing, and (2) encoding/decoding function.

According to embodiments of the present invention employing time domainmultiplexing, an optical shutter (e.g. Liquid Crystal Display shutter)may be included as part of each relevant optical path. Time domainmultiplexing may be achieved by opening only one given optical path'sshutters during a given period, which given period is within anacquisition period during which the image sensor is to produce imagedata associated with the given optical path. By opening the shutter ofeach of the relevant optical paths in correlation with the imagesensor's acquisition periods, multiple image data sets (i.e. imageframes) may be produced, wherein each image data set may be associatedwith a separate optical path.

According to embodiments of the present invention where optical imageinformation received/collected by each of a set of optical paths iscombined (i.e. optically encoded) on a common surface segment of theimage sensor during one or more sensor acquisition periods, the opticalsensor may produce a composite image data set that may includeinformation relating to some encoded composite of the two or morecollected optical images. An optical encoding/decoding apparatus/methodor algorithm in accordance with some embodiments of the presentinvention may be used to encode the two or more collected optical imagesand decode the composite image data set into two or more separate imagedata sets, wherein each of the separate image data sets may beassociated with and may represent collected optical image informationfrom a single optical path.

According to some embodiments of the present invention, there isprovided an image data processing block, implemented either on opticalmeans, a dedicated data processor or on a programmable generable purposeprocessor. One or multiple image processing algorithms may beimplemented or executed via the processing block. For example, inaccordance with some embodiments of the present invention, an opticaldata processing block may be adapted to generate a complexmultidimensional data set to be printed on the camera image sensor. Inaccordance with further embodiments of the present invention a dataprocessing block may be adapted to extrapolate each of the subset ofoptical paths printed on the camera image sensor.

According to some further embodiments of the present invention—theprocessing block may be adapted to combine two or more collected imagesinto a multidimensional (e.g. four dimensional) data set, wherein themultidimensional data set may include image data representing variousfeatures of the common projection plain (i.e. common surface).

According to further embodiments of the present invention, there may beprovided an image extrapolation block, implemented either on opticalmeans, a dedicated data processor or on a programmable generable purposeprocessor. The image extrapolation block may extrapolate either from theoptically encoded complex multidimensional data set, the extrapolatedsubsets of optical paths printed on the camera image sensor (i.e. theoriginally acquired image data sets), or the reconstructedmultidimensional data set, which multidimensional data set may be one ormore types of derived data sets, wherein each extrapolated data set typemay be associated with one or more features of the common projectionplain (i.e. common surface) from which the two or more optical pathscollected optical image information.

According to some embodiments of the present invention, extrapolateddata set types may include: (1) a depth map (i.e. z-channel or depthinformation of every pixel's-point in the common projection plain), (2)a holographic stereogram image of the common projection plain, (3) ahologram image of the common surface, (4) a stereo image of the commonsurface, and (5) one or more two-dimensional images, where each imagemay also be an approximated virtual view point of the common surface.

According to some embodiments of the present invention, each of theoptical paths may include a fixed mirror, a fixed lens and a fixeddiaphragm structure. According to further embodiments of the presentinvention, the lenses and mirrors of two or more optical paths may befunctionally associated (e.g. synchronized). According to yet a furtherembodiment of the present invention, the diaphragms on the two or moregiven optical paths having functionally associated lenses and mirrorsmay be adapted to adjust their configuration (e.g. aperture size andshape) so as to maintain a common projection plain between the givenoptical paths when the focus on the synchronized lenses is changed.

Turning now to FIG. 4A, there is shown an optical assembly according tosome embodiments of the present invention. As visible from FIG. 4A, theoptical assembly has multiple optical paths, which paths may includelenses, mirrors and may also include a diaphragm structure behind eachlens/mirror. The configuration of each optical path's diaphragm, lens,mirror and positioning of the optical sensor may define the projectionplane from where the optical path may receive optical image information.According to some embodiments of the present invention, two or more ofthe optical paths on the optical assembly may be configured to acquireoptical image information from a substantially common surface. Theprojection plain (i.e. surface) for each of the two or more opticalpaths may partially, substantially or totally overlap. The shape of eachoptical path's projection plain may be flat, spherical, cylindrical orany other shape which may be defined using optical elements such aslenses and mirrors.

FIG. 4B shows a diagrammatic representation of an embodiment of theoptical assembly where optical image information from two separateoptical paths are projected onto separate areas of a common imagesensor. According to such embodiments of the present invention, eacharea of the optical sensor may correspond with an image frame. Forexample, if the image sensor outputs an RGB image (may also be in RAWdata) from the entire sensor area, an image processing algorithm mayparse the image into separate frames.

FIG. 4C shows a diagrammatic representation of an optical assemblyaccording to some embodiments of the present invention where opticalimage information from two separate optical paths are projected onto acommon area of a common image sensor. According to these embodiments, adedicated lens, an optical filter and/or a grid associated with eachoptical path may provide for the optical encoding of the optical imageinformation received by each of the optical paths, such that althoughtwo or more optical images may be simultaneously printed onto the samearea of an image sensor, the two images may later be extrapolated fromthe sensor's data set (may also be RAW data) using an image processingalgorithm, according some embodiments of the present invention.

FIG. 5A shows a symbolic diagram of an optical assembly and imageprocessing which according to some embodiments of the present inventionproduces a series of digital image data sets (e.g. 2D image frames).According to some embodiments of the present invention, the frames maybe produced by one or more image sensors. If two or more optical pathsdirect their respective received/collected optical image informationonto a separate area of a common image sensor or onto separate sensors,separate image data sets (i.e. image) frames may be produced directly bythe image sensors.

When separate image frames of a common projection plain aresimultaneously produced through separate optical paths, the separateimage frames may be used to generate a multidimensional image data set,for example a three dimensional (3D) depth map representing theprojection plane. Examples of calculations associated with calculatingdepth information on a pixel by pixel basis (disparity map) are shown inFIGS. 5B and 5C, and a detailed explanation is given herein below.According to the present invention, a color may also be associated witheach of the pixels in the 3D depth map, using one of several methodsaccording to the present invention. The multidimensional image data setmay be extended to yet a further dimension by, for example, usingmultiple sets of images taken at different times to add a time variableto a colorized or non-colorized 3D depth map, thereby producing a 4Ddepth map. According to some embodiments of the present invention, aswill be described below, a 3D depth map may be generated from a singleoptically encoded image data set.

Turning now to FIGS. 5B and 5C, there are shown symbolic diagramsdepicting multiple images and image related data sets being derivedand/or extrapolated from a multidimensional image data set according tosome embodiments of the present invention. As will described furtherbelow, ray tracing, pixel color interpellation and other imageprocessing methods may be used to convert a multidimensional image dataset into a variety of image data types. According to further embodimentsof the present invention, certain image data types may be generateddirectly from acquired image frames.

Turning now to FIG. 6A, there is shown a block diagram showing an imageprocessing element/module according to some embodiments of the presentinvention. The operation of the various elements of FIG. 6A shall bedescribed in conjunction with the steps of the methods illustrated inFIGS. 6B and 6C, where FIG. 6B shows a flow diagram consisting of thesteps of producing one or more multidimensional image data sets from twoor more acquired image frames according to some embodiments of thepresent invention, and FIG. 6C shows a flow diagram consisting of thesteps of producing one or more multidimensional image data sets from anoptically encoded image frame according to some embodiments of thepresent invention.

The received image information may either represent separate imageframes (step 1000B) or RAW data, where each frame is generated fromoptical image information coming from a separate optical path, or thereceived image information may be an optically encoded image data (step1000C) including optical image information from multiple optical pathsmixed onto a common area of an optical sensor.

According to some embodiments of the present invention, the datareceived by element 100 may be directed towards a secondaryextrapolation module and denoted as element 150. the extrapolation unitmay be adapted to extrapolate each of the subset of optical pathsprinted on the camera image sensor (step 1500C), or may be adapted todirectly extrapolate (complex) 2D Image(s) (step 5000B) as explainedherein bellow.

When the received image information is in separate frames, a Depth MapGeneration Module 200 may generate a (3D) depth map (step 2000B) usingone of various depth extraction algorithms, including the point by pointdisparity map calculations as illustrated in FIGS. 1B and 1C.

According to some embodiments of the present invention, the receivedimage information may be optically encoded (step 1000C) such that a (3D)depth map may be derived (step 2000C) without having to performconsiderable calculations, in which case the Interface to Image Sensor(optical processing block) 100 may be adapted to generate the depth map.

A Color Estimation Module 300 may interpolate a color for each of thepoints in the depth map (steps 3000B and 3000C). Various colorinterpolation or estimation methods, including the one described below,may be used.

A 4D matrix generation module 400 may use data associated with multipledepth maps, produced based on images acquired at different times, togenerate a multidimensional image data set (steps 4000B and 4000C) whichincludes time as one of the dimensions.

An image extrapolation module 500 may generate, either from theextrapolated subset of optical paths, in conjunction with the secondaryextrapolation module 150, or from a 4D data set, one or more image datatypes (steps 5000B & 5000C) including simple 2D images from various viewpoints, complex 2D images with encoded depth information, and variousothers which are described below.

The following text contains examples of various algorithms and methodsaccording to some embodiments of the present invention.

The following algorithm enables simultaneously optically encodingtogether on the same single capturing sensor (e.g. CCD) multipledifferent images (up to full sensor resolution for each image), anddecoding said multiple images with out loosing resolution for each ofsaid images. The algorithm may input optically acquired images ormultiple different sources (e.g. cameras, where the encoding may also bedigital).

According to some embodiments of the present invention, the algorithmhigh level may comprise:

-   -   1. Encoding: Using Optical/image processing means.    -   2. Printing images on digital sensor/frame/data stream.    -   3. Input complex data from digital sensor/frame/data stream.    -   4. Decoding complex data, 4D Matrix Reconstruction    -   5. Output

According to some embodiments of the present invention, the Encodingstep may comprise:

-   -   a. Input: multiple optically acquired images (e.g. Holo stereo        capturing device) or input images from multiple sources (e.g.        cameras).    -   b. Encoding said images, using optical means or image processing        means.

According to some embodiments of the present invention, the Printingimages on digital sensor step may comprise:

-   -   a. Simultaneously projecting said encoded images onto a digital        sensor. Or, simultaneously printing said encoded images as a        single frame or any suitable data stream.

According to some embodiments of the present invention, the inputcomplex data step may comprise:

-   -   a. Input images from digital sensor. Or, Input images from data        stream.

According to some embodiments of the present invention, the Decodingcomplex data step may comprise:

-   -   1. Direct access to information in the complex multidimensional        data set,    -   2. Extrapolate each of the subset of optical paths or multiple        images from said encoded data.    -   3. Reconstruct depth maps, 4D dimension matrix    -   4. Reconstruct 3D color.    -   5. Extrapolate complex data sets

According to some embodiments of the present invention, the followingtext describes an example for an encoding decoding reconstruction andcompression Algorithm.

The encoding decoding reconstruction and compression process may be doneon images acquired from multiple optical sources where the encodingprocess is done optically, or from multiple sources such as camerashaving no direct link to the holo-stereo capturing device, where theencoding process is done using image processing tools.

We propose a method and apparatus for “digital holography of incoherentlight encoding and decoding”, reconstructing a 3 dimensional/holographicimage within an N dimensional matrix (for the following examples we willuse a 4D matrix) from optically acquired 2D multiple images or fromordinary cameras (The “input data”), several projections of a unifiedprojection frame (e.g. surface) illuminated by incoherent light, fromdifferent points of view. The encoding process of the input data may bedone by optical means or digital means over space-time, to yield a 2Dcomplex function, which is then optically printed on a digital sensor,or computationally processed as a frame or any other form of datastream.

According to some embodiments of the present invention, the decoding andreconstruction process occurs in two stages. The first stage isdecoding, performed as the quantized complex 2D function image is loadedinto memory, holding a 3D structure or being filtered in to its originalencoded particles (e.g. two images). The second stage, de-quantization,reconstructs the 4D matrix.

The final part of the system is a real time image creator given therequested data and geometry (e.g. a 2D slice of the 4D matrix). This isa real time rendering model.

According to some embodiments of the present invention, image processingalgorithms may also extrapolate data directly from the decoded data(e.g. 3D extraction of a single point in the image or a depth map) andto view one of the decoded images, with no reconstruction of a 4Dmatrix.

In stereogram photography, we record several pictures of a projectionframe (e,g, surface) from different points of view. The encoded complex2D function may be reconstructed into a computer generated holographicstereogram—our 4D matrix, reconstructing the original three dimensionalprojection frame. This means that the image is 3D reconstructed in thevicinity of our 4D matrix, in the horizontal (X) vertical (Y) depth (Z)and time (t) image axis. Consequently, the desired output may beextrapolated to any suitable dedicated means.

To summarize the process, a real world 3-D projection frame is capturedby two or more 2D images from several angles. These images are encodedto a complex 2D function as a single frame. The decoded projections ofthe scene may be computed to a 4D computer generated matrix alsoextrapolating each of said projection angles, holographic stereogram,depth map, depth perception (stereoscopy) and multiple new 2-D matrixprojection angles along X,Y,Z.

4D Matrix Construction and Color Separation methods

A 2D image is a 2D slice of a 4D light field over time. Reconstructing alight field over time from a set of images corresponds to inserting eachof the 2D light field slices into a 4D light field representation overtime—a 4D matrix. Similarly, generating new views, depth values and soon corresponds to real time extracting and resampling a slice ofdifferent data and different views. This requires that the 4D matrixwill properly resample each slice rays intersection representation toavoid distortions in the final image data. This process is the 4D datareconstruction.

Alignment of a pair of images from two adjacent camera/image locationson a 2D plane; a given point will project to different locations,potentially several pixels apart, in these two images. The distancebetween the two projected locations is called the stereo disparity.Extending this idea to multiple camera locations produces a sequence ofimages in which the object appears to jump by a distance equal to thedisparity.

Following this idea, each point in the real world 3D projection frame,holds a certain color we can define as our absolute color, representedin the multiple images on different 2D projections, and following thegiven conditions this absolute color will be overlaid with distortionscreating “color disparity” from the relative points' color and theabsolute color. In the 4D matrix, we will strive to reconstruct thesepoints' absolute color. Different angels exposed to the same point willpreserve this color with given distortions such as angels, differenthorizontal positioning in the image, different lighting, opticaldistortions and so on. But, giving the fact that all projections of thesame point, are a percentage of the absolute color of that point's colorin reality, we can reconstruct each point's color, interpolating thepoint's color from the different angels, target to reconstruct the 3Dpoint color as best as can be to the original absolute color of eachpoint.

This process enables compression of multiple images while preserving thebasic rolls for practical real time compression:

-   -   Data redundancy.—A good compression technique removes redundancy        from a signal without affecting its content. Our method enables        the 3D interpolation of a point from multiple views enabling a        complex 2D function to avoid multiplication of the same data        from multiple images.    -   Random access.—Most compression techniques place some constraint        on random access to data. Predictive coding schemes further        complicate random access because pixels depend on previously        decoded pixels, scanlines, or frames. Our method enables direct        random execs to any given point in space and time.    -   Computational expense.—Our compression scheme of the 4D matrix        enables quickly encoding transferring and decoding said images        without hardware assistance on the resaving side. while        consuming minor computational means.

Exemplary Algorithm

The following example algorithm assumes the case of 3 images taken from3 lenses (FIG. 5A elements 150A, 150B and 150C) of the holo-stereocapturing device; a single CCD (some cameras have 3CCD array one foreach color or CMOS) and Bayer filter (FIG. 3).

The encoding methods may alter pending on the overall settings (inputimages, optical configurations, 3CCD, CMOS, fps and so on.). But itshould be realized to those skilled in the art that it does not changethe process of the encoding, decoding, reconstruction and compressionalgorithm.

According to some embodiments of the present invention, anencoding/decoding reconstruction and compression algorithm may comprisethe following steps and may be described in conjunction with FIG. 10 andFIG. 5A:

1. Optical Encoding

1.1 Input: multiple optically acquired images (FIG. 10 step 100, imagesmay be acquired using elements 150A, 150B and 150C shown in FIG. 5A,wherein the common projection plane is denoted as element 100 of FIG.5A).

-   -   Following the process of optical acquisition, as described        earlier, the holo-stereo device now optically holds multiple        images in time t′.

1.2 Optically Encoding said images, using optical means (FIG. 10 step200, the encoding may take place in element 200 (Optical processingblock) in FIG. 5A).

1.3 Creation of a 2D complex function that holds the information fromthe multiple optically acquired images (FIG. 10 step 300, element 300 inFIG. 5A)

1.4 Extrapolate Subsets of optical complex multidimensional data set(FIG. 10 step 400, the extrapolation may take place in element 400(Image processing Block) of FIG. 5A)

The Encoding algorithm (i.e. optical process) itself may vary (e.g.encoding using color, Fourier, Diffraction, Fresnel, Spatial etc), andit is not limited to specific configuration, the following example isnot a limiting one.

Following are Three Exemplary Optical Encoding Methods—

Denote:

Image 1=I1, its color: R1G1B

Image 2=I2, its color: R2G2B2

Image 3=I3, its color: R3G3B3

The encoding in this example may be the following:

Method 1:

-   -   Projecting the images through dedicated filters such that the        green color will enable the disparity maps reconstruction. Since        it is best to have the same color to enable disparity maps of        multiple images (different colors from different images may not        be accurately matched), and since the CCD holds ½ amount of        Green, ¼ red ¼ blue, and disparity in horizontal parallax is        done on horizontal vectors, we will use the green in the rows to        enable initial disparity maps. The encoding may be in the form        of projecting said input images through optical filters such        that said images are projected on said CCD as shown in FIG. 7A,        Line 1 (7100), Line 2 (7140) and Line 3 (7170), wherein red        color is denoted by “R”, blue color is denoted by “B”, and green        color is denoted by “G”.    -   Some color points are missing on different images, their        signature can be printed on the digital sensor using the physics        attributes of the digital sensor. We may also use different        spatial deviation on the CCD.

Method 2:

-   -   Enhancing said images optical signature on the digital sensor        using the reaction of each CCD receptor to light, or the amount        of data each pixel may hold in a single frame or any means of        data transfer.    -   The numbers of photons in each photo sensor on the CCD may        interpreted to digital values from 0 to 2̂12 using 12 bit for        each sensor in some RAW files and even higher bit for each        sensor.    -   Method 2 exploits the potentially larger bandwidth of each photo        sensor in terms of digital light printing, as if to exploit the        total CCD information more then the usual output value of the        pixel (mostly 8 bit per pixel per each color). Using filters to        optically vary the values of image points during the projection        step on the CCD sensor, larger values may be collected and later        interpreted to digital terms in a 12 bit array or more.        Understanding the idea that the image sensor can be looked at as        an optical memory unit, where values can be analyzed as digital        values of a memory unit. Using higher storage values and optical        encoding manipulations, one can encode and extrapolate number of        values to be decoded as different images.    -   One example for this method is given bellow.

Method 3:

-   -   Projecting the set of optical paths through dedicated optical        means, for example by diffraction. The set of optical paths in        the formation of Fresnel and/or Kirchhoff and/or Frauenhoffer        diffraction are convoluted and recorded on a camera image sensor        and stored digitally.    -   The decoding and reconstruction is performed by numerical means        based on the use of fast Fourier transform (FFT), and may also        use spatial and/or temporal filtering as well polarization        means.

2. Printing Images on Digital Sensor:

-   -   Simultaneously projecting said encoded images on digital sensor.

3. Input Image from Digital Sensor:

-   -   Input images from digital sensor may also be in a 12 bit RAW        format, and may also be prior to a color interpolation processes        (e.g. Demosaicing).

4. Decoding Complex Data, 4D Matrix Reconstruction.

4.1 Decoding Complex Multidimensional Data Set Process

-   -   is done in order to extrapolate each of the subset of optical        paths printed on the camera image sensor. Multiple data sets may        be extrapolated directly from the complex multidimensional data        set and/or from the extrapolated subset of optical paths.        Following the decoding process is the 4D matrix reconstruction.    -   In the case of spatial encoding images might be reconstructed by        reading the dedicated pixels from their printed coordinates on        the CCD, thus reconstructing a subset of optical paths.        Diffracted optical paths in the formation of Fresnel and/or        Kirchhoff and/or Frauenhoffer diffraction that are convoluted        and recorded on a camera image sensor and stored digitally.    -   The decoding and reconstruction is performed by numerical means        based on the use of fast Fourier transform (FFT), and may also        use spatial and/or temporal filtering as well polarization        means.    -   Complex multidimensional data sets based on color interpolation        may be extrapolated by spatial and/or time decoding, by digital        values interpolation as in the example of method 2, and in a 3D        reconstruction process

4.2 Reconstruct Multiple Images' Depth Maps in a 3D Matrix (XYZ)

-   -   Following the process of encoding the complex multidimensional        data set, multiple data sets may be extrapolated directly from        the complex multidimensional data set and/or from the        extrapolated subset of optical paths (e.g. extrapolate depth        maps of the input images), or to reconstruct a unified 3D image        in a 4D matrix. In the following example the subset of optical        paths of the complex multidimensional data set is the input for        the 3D structure (depth reconstruction) placed in the 3D matrix.        Depth extraction using image color information is best performed        using the same color of the different images. Different colors        might not extrapolate proper depth maps (even if we convert them        to gray scale for example), because different color acquisition        print different visual signatures on the image (such as a red        table will leave low green or blue signatures if any). Since in        the following not limiting example, we are using a single CCD        with Bayer filter, the green holds half of the sensor original        points, and since the depth map is done on the rows of the image        to check the parallax disparity between the images, we need to        enable a row of green for the first 11 camera (the left one),        and on the same row, a row of green for the second image 12, to        enable depth calculation.

Turning now to FIG. 7B, there is shown the exemplary filter describedhereinabove, the first encoded row/line is denoted by 7200. We willseparate each element (G1/G2/R1) and receive the filters denoted by7210, 7220 and 7230.

It should be clear to one of ordinary skills in the art that the Redresolution of image 1 was not damaged.

The outcome of reconstructing the vector color of each image fordisparity purposes is shown in FIG. 7C, wherein line 1 is denoted by7300, and line 2 is denoted by 7310.

We will use these rows to create a disparity map, depth map of the firstrow of images 1 & 2 using previously dedicated algorithms as previouslydiscussed and as will further be explained by example algorithms, andplace the registrated rows in a 3D matrix. The 3D rays' intersection ofthe original 3D projection frame (surface) are aligned.

Having positioned the points in 3D, we can now reconstruct the greenvector color of row one & two using projected green from image 1, 2. Andfollowing the same process on lines 2 & 3, we will enable a properdisparity map for images 2 & 3, as seen in 7330.

The positioned points in 3D, may enable us to reconstruct the green (orfull RGB) as will further be explained to rows 1, 2, 3—images 1, 2, 3,as seen in FIG. 8

4.2 Reconstruct 3D Color in the 3D Matrix.

The reconstruction of RGB for 3D point p′ is performed in the followingnot limiting example: following the fact that we aligned in 3D the twoimages we now have for example the red color from image 1-line 1, asexplained earlier, its resolution was not damaged, we also have the bluecolor of the same example point p′ from image 2 line 2, blue colorresolution was also not harmed, and since we also reconstructed thegreen in both rows, we can now interpolate and reconstruct the RGB forpoint p′ based on the 3D location of neighboring pixels rather then ontheir original 2D position on the CCD in the first and second line.

The major difference of 3D color reconstruction in the 3D matrix withrespect to the 2D RAW file Demosaicing, is the 3D positioning of eachpoint in the 3D matrix, reconstructing the 3D structure of theprojection frame in the real world,

Rays intersect in 3D, as opposed to their 2D location in the 2D image,each point in the 3D matrix resaves color from at list two differentview points. In our example, in line 1 point Pi receives red and greenfrom image 1 and green from image 2, in the 2nd line receives green fromimage 1, green and blue from image 2, and so the ratio of ½ amount ofgreen, ¼ amount of R and ¼ amount of B is kept for each point, the 3Drays intersection allows for accurate positioning of 3D color, and the3D Demosaicing will now take place, reconstructing a 3D color point inspace, the Demosaicing is based on the 3D neighboring pixels from eachimage point of view up to a (neighboring) trash hold, fullyreconstructing the image lines, up to a full array of 12 bit (or more)per point of red, full array of 12 bit (or more) per point of blue, andsame for green.

Distortions on the reconstructed color will be 4D filtered over spaceand time.

4.3 Extrapolate Full RGB for Each of Input Images and Arbitrary CameraPositions

-   -   since we now hold a XYZt′, RGB-3D color for each point in the 4D        matrix, we can generate 2D images also from arbitrary camera        positions with depth information. An image is a 2D slice of the        4D light field.

The pervious example was compression over space, where compression overtime may also be generated using said encoding in the 4D matrix, addingthe temporal vector to the XYZ spatial coordinates and RGB.

If the compression algorithm is used for compression of data transfer,the 4D matrix may be done only once, on the compressing side. Followingthe 4D matrix digital reconstruction, a new 2D complex function might begenerated that encapsulates each pixels, RGB, XYZ, where the Z valuemight be encapsulated in the image data by, for example as previouslymentioned in method 2, using higher bit values for each pixels (needonly one extra value to RGB XY—the Z), enabling to immediately decodethe 4D image on the receiving side with very low computation needs.

Following is a simple numerical example to the 3D reconstruction andcompression algorithm, using image processing means (i.e. the input aretwo images). The image pixels are represented in their digitalrepresentation, enabling to accurately value the compression process,and also in a printed version.

The input is 2×2D images. The algorithm comprises the steps of: (1)reconstruct a 3D matrix, (2) encode said matrix to a 2D complexfunction, enabling computation less decoding on the receiving end.

Turning now to FIG. 9A there is shown two input images for this example,the left image is denoted by 9100 and the right image is denoted by9200. The first step is extracting a depth map. The input images arefull array of RGB each, accordingly a depth map can be reconstructed aspreviously explained on all image layers:

The end process is the positioning of pixels in a 3D space, intersectingrays from both images.

Turning now to FIG. 9B there is shown a region positioned on the samedepth, of the Mobile phone that is located on the right of the white capof coffee, the left image is denoted by 9300 and the right image isdenoted by 9400

A vector of pixels on the same depth of one color from two images wouldlook like that:

R 42 42 33 21 13 14 28 58 97 144 176 L 45 42 36 30 18 17 26 51 95 138179

Having positioned every pixel in the image in a 3D space we can unifyits color by interpolating the intersected rays, encapsulating them into a single pixel. The interpolated image may be the following example(3XMatrix layers for Red Green and Blue

${M\; 2\left( {:{,{:{,1}}}} \right)} = \begin{matrix}0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \\0 & 43 & 42 & 34 & 25 & 15 & 15 & 27 & 54 & 96 & 141 & 177 & 0 \\0 & 38 & 34 & 25 & 18 & 12 & 16 & 38 & 71 & 117 & 165 & 204 & 0 \\0 & 32 & 23 & 14 & 9 & 14 & 30 & 59 & 100 & 146 & 192 & 227 & 0 \\0 & 28 & 15 & 5 & 7 & 19 & 43 & 84 & 133 & 176 & 217 & 240 & 0 \\0 & 23 & 11 & 7 & 11 & 29 & 61 & 114 & 167 & 205 & 230 & 250 & 1 \\0 & 18 & 11 & 11 & 25 & 51 & 89 & 146 & 196 & 226 & 244 & 249 & 0 \\0 & 16 & 11 & 20 & 47 & 84 & 130 & 182 & 222 & 244 & 251 & 247 & 1 \\0 & 16 & 20 & 38 & 72 & 118 & 164 & 211 & 238 & 249 & 251 & 242 & 0 \\0 & 19 & 35 & 63 & 101 & 147 & 191 & 225 & 245 & 250 & 242 & 227 & 0 \\1 & 30 & 57 & 92 & 130 & 173 & 212 & 235 & 243 & 241 & 226 & 209 & 0 \\0 & 47 & 84 & 125 & 162 & 196 & 222 & 232 & 234 & 224 & 207 & 188 & 0 \\0 & 65 & 107 & 149 & 183 & 210 & 225 & 229 & 222 & 210 & 190 & 171 & 0 \\1 & 83 & 122 & 160 & 190 & 211 & 224 & 221 & 210 & 198 & 178 & 161 & 0 \\1 & 101 & 139 & 171 & 196 & 209 & 212 & 206 & 194 & 182 & 171 & 155 & 2 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}$ ${M\; 2\left( {:{,{:{,2}}}} \right)} = \begin{matrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 44 & 45 & 37 & 27 & 18 & 18 & 31 & 59 & 103 & 149 & 183 & 1 \\0 & 39 & 36 & 28 & 20 & 17 & 21 & 44 & 77 & 124 & 172 & 211 & 0 \\0 & 33 & 24 & 15 & 13 & 18 & 34 & 65 & 107 & 154 & 198 & 235 & 0 \\0 & 28 & 16 & 9 & 10 & 24 & 49 & 92 & 140 & 184 & 223 & 248 & 0 \\0 & 23 & 14 & 9 & 14 & 32 & 66 & 118 & 171 & 211 & 237 & 253 & 0 \\0 & 19 & 13 & 13 & 28 & 52 & 91 & 148 & 201 & 231 & 250 & 252 & 0 \\0 & 16 & 13 & 23 & 50 & 86 & 131 & 184 & 226 & 246 & 254 & 250 & 0 \\1 & 18 & 22 & 40 & 74 & 120 & 167 & 213 & 241 & 252 & 254 & 244 & 0 \\0 & 22 & 37 & 64 & 102 & 149 & 193 & 224 & 245 & 250 & 243 & 227 & 0 \\0 & 31 & 59 & 93 & 132 & 175 & 213 & 234 & 244 & 242 & 227 & 210 & 0 \\0 & 48 & 85 & 127 & 164 & 198 & 224 & 232 & 234 & 223 & 207 & 189 & 0 \\0 & 66 & 108 & 151 & 184 & 212 & 227 & 230 & 224 & 209 & 188 & 173 & 0 \\0 & 84 & 124 & 162 & 193 & 213 & 226 & 219 & 210 & 195 & 176 & 162 & 0 \\0 & 102 & 141 & 172 & 197 & 210 & 214 & 205 & 193 & 179 & 166 & 154 & 0 \\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0\end{matrix}$ ${M\; 2\left( {:{,{:{,3}}}} \right)} = \begin{matrix}0 & 1 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 45 & 44 & 38 & 30 & 19 & 20 & 32 & 61 & 104 & 150 & 184 & 4 \\0 & 40 & 36 & 30 & 23 & 19 & 23 & 45 & 77 & 125 & 174 & 212 & 2 \\0 & 35 & 26 & 18 & 15 & 20 & 37 & 65 & 108 & 156 & 201 & 236 & 1 \\0 & 31 & 18 & 12 & 12 & 26 & 51 & 91 & 140 & 186 & 226 & 248 & 0 \\0 & 26 & 17 & 13 & 17 & 36 & 69 & 120 & 172 & 213 & 239 & 253 & 0 \\0 & 21 & 16 & 15 & 30 & 55 & 94 & 149 & 203 & 233 & 251 & 253 & 1 \\0 & 19 & 15 & 24 & 50 & 86 & 133 & 182 & 222 & 241 & 248 & 244 & 0 \\1 & 19 & 23 & 40 & 74 & 120 & 166 & 209 & 235 & 246 & 244 & 237 & 0 \\1 & 23 & 38 & 67 & 104 & 149 & 193 & 220 & 240 & 241 & 232 & 218 & 2 \\1 & 33 & 59 & 94 & 132 & 175 & 212 & 229 & 237 & 231 & 216 & 199 & 2 \\0 & 48 & 84 & 126 & 161 & 194 & 220 & 224 & 226 & 216 & 197 & 179 & 0 \\2 & 68 & 108 & 150 & 182 & 209 & 223 & 223 & 214 & 201 & 179 & 161 & 0 \\0 & 83 & 121 & 158 & 187 & 206 & 220 & 211 & 202 & 184 & 167 & 149 & 0 \\0 & 100 & 136 & 169 & 191 & 206 & 211 & 199 & 185 & 167 & 157 & 142 & 0 \\1 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 4\end{matrix}$

The graphical outcome of the above explained manipulations is depictedin FIG. 9C.

Printing the image pixels as a single image, we would also want to addtheir Z value so that the receiving end of the data will not needcomputation. Using mathematical manipulations similar to the secondpreviously mentioned method, The outcome 2D complex function would beXYZ RGB, for every pixel in the image.

5. Output

5.1 Output from the 4D Matrix

Fast generation of different data and views, is generated by rendering a2D array of image data, wherein the 2D slice of the 4D matrix representsrays through a point, properly resampled to avoid artifacts in the finalimage data.

5.1.1 Holographic Stereogram (still & Video):

5.1.2 Extrapolate multiple discrete dedicated view points to HolographicStereogram display device.

5.1.3 Using digital holography (e.g. Fourier transform and Fresneldiffraction as previously discussed) to digitally project said 4D matrixas a 2D image for digital holography future reconstruction.

5.1.4 Depth map: extrapolate Z channels of the 4D matrix.

5.1.5 Depth Perception: extrapolate stereo discrete view points tostereoscopy display device.

5.1.6 2D images (from multiple view points): extrapolate a discrete viewpoint to suitable display device.

5.2 Output following the decoding process of the complexmultidimensional data set without 4D matrix reconstruction.

The optically encoded complex multidimensional data set holds the 3Dattributes of the photographed projection frame (as previouslyexplained).

5.2.1 One can access these attributes and extrapolate the informationdirectly from the 2D complex function.

5.2.2 The encoding process may also enable extrapolation of, forexample, the multiple 2D images of the projection frame.

5.2.3 Once the multiple 2D images of the projection frame are decoded,the images may be viewed directly as 2D images of the projection frame.

5.2.4 Depth information of desired 2D points, image parts, and fulldepth maps using passive methods may be extrapolated, as will further beexplained, from the multiple images of the projection frame.

5.2.5 One can also output the multiple images for holographic stereogramand stereoscopic display.

5.2.5.1. Using images from each slit for viewing, a viewer looking atthe stereogram will see two different projection views through two slitapertures, one through each eye. The brain interprets the differencesbetween the two views as three-dimensional information. If the viewermoves side to side, different pairs of images are presented, and so thescene appears to gradually and accurately change from one viewpoint tothe next to faithfully mimic the appearance of an actualthree-dimensional scene.

5.3 Stereographic and Stereoscopic Display Algorithm

Processing algorithm for creating holographic-stereogram andstereoscopic images from said collection of images may also be thefollowing process, wherein said holographic-stereogram and stereoscopicimage pair covers said holo-stereo lens Projection plane—POV (point ofview).The image captured by each mini-lens/slit is processed separately.Distorted information will be removed from each image of each lensseparately. The images are then cropped into pre-defined sections whichare required for keeping an adequate overlap between adjacent images.For stereogram and stereoscopic display, a stereoscopic image pair iscreated from the group of selected images in the following non limitingexamples:

-   -   Each of the selected lens images is divided into a left part and        a right part using a line passing through the center of the        image.    -   All the left parts generated are merged into one uniform two        dimensional image for the right image of the stereoscopic pair        to be displayed to the viewer's right eye.    -   Following the same line, a left image is formed by merging        together the right parts generated from the images.        A stereoscopic image pair can also be created from the group of        selected images in the following non limiting example:    -   Each of the slits/mini-lens being on the left side of two        adjusted images is divided into a left lens.    -   Each of the slits/mini-lens being on the right side of two        adjusted images is divided into a right lens.    -   All the left lenses are merged into a one uniform two        dimensional image for the left image of the stereoscopic pair to        be displayed to the viewer's left eye.    -   Following the same line, a right image is formed by merging        together the right lenses.

The new images Left and Right both cover the viewer's field of visionbut obviously they are not identical, and are perceived as were takenfrom two different viewpoints, thus giving the viewer a stereoscopicperception.

Roughly, this new image pair is equivalent to a pair of images taken bytwo virtual lenses having their optical axes directed forward in theviewer's viewing direction and having horizontal disparity.

The same holds using any of the Holo-Stereogram optical configurationswherein the field of vision of each lens overlaps to a great extent thefields of view of all adjacent lenses and the lenses are optionallyequally distributed on the lens's surface.

Using the configuration where multiple mini lenses are distributed onthe Holo-Stereogram lens surface, stereoscopic image pairs can also becreated when the viewer's horizon is inclined with respect to ground(i.e., when the viewer eyes are not at the same height with respect toground). For such cases, the selection horizon of the images is done byprojecting the viewer's horizon on the collection of the lenses and thestereoscopic pair is generated by following same steps.

Displaying

The data and images may be displayed to a viewer in various formats,such as stills, video.

The images formed can be displayed by any of the dedicated means such asdedicated hologram display device, stereoscopic viewing, virtualreality, and so on. Depth maps can be viewed in a display devicesuitable for 3D viewing, can be exported to any 3D image or graphicimage processing software to be used for editing and interpolation ofany kind using the 3D extracted information.

Image Processing Manipulations of the Images

The process of image processing manipulations of the images is comprisedof endless possible options for image manipulations in real time basedon the 3D information of the images.

The 3D images are a virtual world. Said virtual world is created in adesignated area, where every point in the camera's POV is a 3D point inthe virtual world. In the said virtual world 2D or 3D real images andcomputer generated images from different sources may be merged.

The 3D position of every point in this virtual world is known. We cansynchronize between virtual world and other photographed images orcomputer generated images using space and time synchronization, fittingreal world images and computer generated images in the virtual world inspatial and temporal terms.

For those skilled in the art, said virtual worlds are referred to as avirtual studio for example, using the 3D information from the virtualworld. Manipulations such as separation between a figure and itsbackground, based on the distance of the figure from the camera arepossible. Isolating a figure from its surrounding we can interlace thefigure in the virtual world created in a computer.

The opposite thing can also be done interlacing CGI figures orphotographed images from different sources in the 3D image.

-   -   Holographic stereogram and stereoscopic information enables        enhancement of the 3D sensation to the viewers interpolating not        only on the 3D positioning in the image, but also in the 3D        sensation using stereoscopic imagery.

The present invention is not limited with regards to the type ofcamera(s) used for capturing the images or sequences of images. Thecamera(s) may be selected from any known in the art digital or analogvideo/still cameras, and film cameras. If needed, non-digital data maybe converted to digital using known in art techniques.

According to some embodiments of the present invention, a holographicstereogram can also be cylindrical or spherical for panoramic orspherical view. This is achieved by producing stereograms with cylinderor spherical projection planes.

Depth Map Reconstruction

The depth map reconstruction process computes the depth map of aprojection plane captured by the holographic stereogram capturing deviceimages, from multiple view points, reconstructing the 3D formation ofthe projection frame as a 4D matrix. The collective field of visioncaptured by all said images covers the whole projection frame of saidarray of images, and wherein any point in said projection frame iscaptured by at least two of said images.

The first part of the algorithm may be considered in the case whereinput images from the digital sensor are full RGB images. The printing(e.g. encoding) methods on the digital sensor and reading said images(e.g. decoding) as full RGB images were described earlier.

Depth Map Reconstruction Method

The following example will demonstrate a basic depth map reconstructionmethod. It should be realized to those skilled in the art that the 4DMatrix algorithms are not limited to any such specific method.Traditional depth map reconstruction methods match points and vectors inthe spatial domain to determine correspondence between pixels inmultiple images in a single static moment in time. Correspondingfeatures are identified, and rays are intersected to find the 3Dposition of each feature.

Adding the time domain, using multiple frames across time, we match asingle pixel from the first image against the second image. Rather thanincreasing a vector by considering a neighborhood in the spatialDirection, it is possible to increase a vector in the temporaldirection. Space-time depth map reconstruction adds a temporal dimensionto the neighborhoods used in the matching function. The computationmatches based on oriented space-time windows that allow the matchingpixels to shift linearly over time. The match scores based on space-timewindows are easily incorporated into existing depth map reconstructionalgorithms. The matching vector can be constructed from an arbitraryspatiotemporal region around the pixel in question. In the case ofrectangular regions, a window of size N×M×T can be chosen, Where N and Mare the spatial sizes of the window, and T is the dimension along thetime axis. The optimal space-time matching window depends on the speedswith which objects in the scene move. Static scenes—a long temporalwindow will give optimal results. Scenes with quickly moving objects—ashort temporal window is desirable to avoid the distortions when objectsmove at intermediate speed, it is likely that a space-time matchingwindow with extent in both space and time will be optimal.

A correspondence must exist between the holographic stereogram exposuregeometry and the 4D Matrix reconstruction.

The following example is a Pseudo Code for depth map reconstruction:

VARIABLE DEFINITIONS

-   -   f=is the focal length.    -   B=is the distance between the cameras.    -   X₁, X₂=the location of point P in Image 1 and in Image 2.

Assumptions:

Since the difference between Z1 and Z2 (which are the Z value of point 1and point 2) depends only on the differences of the X values, we cangenerate Z which is not absolute, but is relative to one of the pointsin the image.

Pseudo Code (for Input of 2 Images)

-   -   Take some point p find its x₁ and x₂, as an example, one can use        SSD (sum of square difference).    -   Denote the expression (x₁−x₂) for point p: “deltaX[p]”    -   Z of point p (Zp) will be: f*B/deltaX[p]    -   take another point p+1, compute “deltaX[p+1]”    -   Z of p+1 (Z[p+1]) will be: f*B/deltaX[p+1]    -   K=deltaX[p+1]/deltaX[p]    -   So Z[p+1]=f*B/deltaX[p+1]=(f*B)/(K*deltaX[p])=Z[p]/K    -   Using this scheme Z[p+n] every point p+n (while n=1 . . . ) we        can write using Z[p] And some known number K (K can differ)    -   Then find n=m such that K for his point [p+m] will be the        biggest And so Z[p+m] will be the smallest.    -   Define Z[p+m]=1, and find Z[p] from here.    -   Then find the values of all Z[p+n] where n=1.

Input of Multiple Images:

X1(w)=(f·x)/z and Xk(w)=f(X−b1k)/z

-   -   Where K=2, . . . . , N    -   b1k=        O1        Ok—in the baseline corresponding to the Kth view point    -   f—Focal distance    -   with respect to first (leftmost) image I1    -   each point W is viewed in the other images with respective        disparities

dk(w)={Xk(w)−X1(w) if W is visible in Ik

{undefined if W is not visible in Ik (is occluded))

-   -   where K=2, . . . , N    -   If W is visible in Ik so:

dk(w)=−f(b1k)/z

-   -   the end disparity of W point−d (w) will be

d(w)=(Σk=2N dk(w))·1/(N−1)

The output of such an algorithm is a depth map, located in a 4D matrix,corresponds to the multiple images of the holographic stereogram, alongthe time axis.

3D Color Reconstruction

The next step will be the 3D color reconstruction. The followingexemplary 3D Color reconstruction method may be used in the case wherethe input multiple images are full RGB each, where each point in theprojection plane is viewed from number of viewpoints, a finite number ofdifferent views of the scene. The outcome display of theholo-stereography is continuous 3D color information of an object'sappearance, approximated by a finite number of two-dimensional images ofthat object. The 3D color reconstruction is the horizontal approximationof the continuous natural color captured optically in a discrete form.The output images from the 4D matrix are images of natural scenes withnatural lighting.

The reconstructed image in the 4D Matrix hold full RGB color for everypoint in the image from every point of view. Pending on the dedicatedoutput, one way of output would be to leave the color as layers of fullcolor from every exposed point of view, outputting the image from adiscrete point of view with its original color. The other way would beto re-interpolate each point into unified color information of multipleview points. This is especially important in digital sensors such as CCDand Bayer filters where the ¾ of the red and blue, and half of the greencolor in every point in the image are reconstructed from surroundingpoints and thus re-demosaicing can enhance dramatically the imagequality. Known in the art methods exist to interpolate color for everypoint. In the 4D matrix, re-interpolation will overcome imagedistortions, enhance image quality and enable higher resolution. Ageneral example is given here. It should be realized to those skilled inthe art that the 3D color reconstruction algorithm is not limited to anysuch specific method.

The camera's horizontal position determines the angle at which thecamera's rays strike the plane of the hologram. Viewing the input imagesin 2D, similar points in the projection frame appear to be located inthe image in deferent horizontal positions due to the disparity of theimages exposed to the projection frame. So color reconstruction cannotbe done on points that are located on the similar (X,Y) 2D coordinatesin the different images. Following the process of the 4D matrix, theprojection frame is reconstructed in a 3D matrix and similar points areidentified in the different images and in the 3D matrix, each point fromevery image is located in the identical points' location seen from thesaid images.

Since the point's color in the real world is a color we can define asour absolute color, the one we will strive to reconstruct in the 3Dmatrix, different angels exposed to the same point will preserve thiscolor with given distortions such as angels, different horizontalpositioning in the image, different lighting, optical distortions and soon. But, given the fact that all projections of the same point are apercentage of the absolute color of that point's color in reality, thereis no point in preserving each point's color from different angels butrather to interpolate the point's color from the different angelstargeting to reconstruct as best as can be the original absolute colorof each point.

furthermore, demosaicing based on 3D location of points in space,interpolate color of points based on surrounding points in the samedepth, prevents many distortions that interpolation in 2D images sufferfrom, for example we interpolate the edge of a wooden table pixel withthe pixels of the wall behind him, as opposed to interpolate the woodonly with points that surround him in 3D, giving higher image quality toevery point in the image and enhancing resolution.

A demosaicing algorithm is used to interpolate a set of RGB color forevery point in the image also enabling enhancement of resolution (addingmore interpolation points then in the original images), using thepoint's color as projected from the different viewing angles, and theneighbors 3D surrounding each point, reconstructing a final image whichcontains full color information (RGB) at each pixel. This process may bedone using existing methods to obtain better interpolation.

The demosaicing algorithm works as a RAW format in the sense that itcontains pixel information far greater then the final outcome, targetedto reconstruct the absolute color on the one hand while preserving thecolor characteristics from every angle on the other, in exporting thedata or reconstructing new view points that where not exposedoriginally.

The computer generated 4D matrix of the holographic stereogram is thereconstructed outcome of the non coherent light digital holographyapparatus (i.e. this invention). Computer generated holography (CGH) mayalso be digitally printed using known in the art computer generatedFresnel and/or Kirchhoff and/or Frauenhoffer diffraction by simulatingcomputer generated coherent light.

In this invention, the process of capturing, optically encoding thecomplex multidimensional data set, decoding and reconstruction of the 4Dmatrix process under non coherent illumination and digital computing,may be the equal process of coherent light digital holography and/orstereographic hologram process.

According to another embodiment of the present invention,

The optical encoding process that creates the complex multidimensionaldata set equals the complex phase and amplitude digital printing andreconstruction under coherent light. In this embodiment, convolveddiffracted light (from the object) propagates through a particularoptical system, thus succeeds in recording the complex amplitude of somewave front without beam interference. The claim is that complexamplitude can be restored under non coherent conditions. Once thiscomplex function is in computer memory, one can encode it to a CGH(computer-generated hologram).

This CGH may then be illuminated by a plane wave, which then propagatesthrough the proper optical system. The reconstructed image has featuressimilar to those of an image coming from a coherently recorded hologram.

While certain features of the invention have been illustrated anddescribed herein, many modifications, substitutions, changes, andequivalents will now occur to those skilled in the art. It is,therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the true spiritof the invention.

1. An apparatus for acquiring image data sets of a given scene via twoor more optical paths, said apparatus comprising: an image sensor arrayincluding first and second sets of light sensing elements; and at leastfirst and second optical paths directing light from the given scene, viadifferent optical paths, onto said image sensor, wherein imageinformation from said first optical path is received by sensing elementsof said first set of light sensing elements and image information fromsaid second optical path is received by sensing elements of said secondset of light sensing elements, and wherein at least one of said sensingelements of said first set of light sensing elements is situated atleast partially between two or more of said sensing elements of saidsecond set of light sensing elements.
 2. The apparatus according toclaim 1, further comprising an image extrapolation block for deriving,from the image information received from said at least first and secondoptical paths, image data sets representing the given scene from singleoptical paths.
 3. The apparatus according to claim 2, wherein the singleoptical paths include optical paths other than said at least first andsecond optical paths directing light from the given scene.
 4. Theapparatus according to claim 1, wherein each of said at least first andsecond optical paths comprises a distinct optical element.
 5. Theapparatus according to claim 4, wherein said distinct element is adistinct filter.
 6. The apparatus according to claim 4, wherein saiddistinct element is a distinct grid.
 7. An apparatus for acquiring imagedata sets of a given scene via two or more optical paths, said apparatuscomprising: an image sensor array including first and second sets oflight sensing elements; and at least first and second optical pathsdirecting light from the given scene, via different optical paths, ontosaid image sensor, wherein image information from said first opticalpath is received by at least one sensing element of said first set oflight sensing elements, which at least one sensing element of said firstset of light sensing elements is situated between two or more of saidsecond set of light sensing elements, which two or more of said secondset of light sensing elements receive image information from said secondoptical path.
 8. The apparatus according to claim 7, further comprisingan image extrapolation block for deriving, from the image informationreceived from said at least first and second optical paths, image datasets representing the given scene from single optical paths.
 9. Theapparatus according to claim 8, wherein the single optical paths includeoptical paths other than said at least first and second optical pathsdirecting light from the given scene.
 10. The apparatus according toclaim 7, wherein each of said at least first and second optical pathscomprises a distinct optical element.
 11. The apparatus according toclaim 10, wherein said distinct element is a distinct filter.
 12. Theapparatus according to claim 10, wherein said distinct element is adistinct grid.
 13. An apparatus for acquiring image data sets of a givenscene via two or more optical paths, said apparatus comprising: an imagesensor array including first and second sets of light sensing elements;and at least first and second optical paths directing light from thegiven scene, via different optical paths, onto said first and secondsets of light sensing elements respectively, such that image informationfrom each of said first and second optical paths is received by one ofsaid first and second sets of light sensing elements, and wherein atleast one sensing element of said first set of light sensing elements issituated between two or more sensing elements of said second set oflight sensing elements.
 14. The apparatus according to claim 13, furthercomprising an image extrapolation block for deriving, from the imageinformation received from said at least first and second optical paths,image data sets representing the given scene from single optical paths.15. The apparatus according to claim 14, wherein the single opticalpaths include optical paths other than said at least first and secondoptical paths directing light from the given scene.
 16. The apparatusaccording to claim 13, wherein each of said at least first and secondoptical paths comprises a distinct optical element.
 17. The apparatusaccording to claim 16, wherein said distinct element is a distinctfilter.
 18. A method for acquiring image data sets of a given scene,said method comprising: directing light from the given scene onto afirst set of light sensing elements of an image sensor array, via afirst optical path; substantially concurrently, directing light from thegiven scene onto a second set of light sensing elements of the imagesensor array, via a second optical path distinct from the first opticalpath; receiving image information via said first optical path by saidfirst set of light sensing elements; and substantially concurrentlyreceiving image information via said second optical path by said secondset of light sensing elements, which second set of light sensingelements is distinct from said first set of light sensing elements;wherein at least one sensing element of said first set of light sensingelements is situated between two or more sensing elements of said secondset of light sensing elements.
 19. The method according to claim 18,further comprising using processing circuitry to derive, from the imageinformation received via said first and second optical paths, image datasets representing the given scene from single optical paths.
 20. Themethod according to claim 19, wherein the single optical paths includeoptical paths other than said first and second optical paths.